About The EITS Workflow Software

A workflow is a term to describe the sequence of steps required to meet the processing requirements of a particular workload.

Because AVED processing is computationally expensive, and the video must be converted to various formats to make it suitable for analysis, a distributed network of computers to manage data storage, video transcoding, and AVED processing is required. To manage the EITS workflow, we use the workflow management and batch job management tools Condor DAGMan. Condor provides many capabilities useful for the experiment including:

• Scaleable computing
  • Install AVED and Condor on as many machines as needed
  • Jobs are matched to the available machines
• Resource management
  • Memory and CPU load monitoring
  • Configure policy to only run if resource available
• Checkpoint mechanism
  • Take snapshot of current program state in such a way that the program can be restarted from that state at a later time
  • Allows for preemptive-resume scheduling

About The Condor and Pegasus tools

From the website http://www.cs.wisc.edu/condor/

Condor is a specialized workload management system for compute-intensive jobs. Like other full-featured batch systems, Condor provides a job queueing mechanism, scheduling policy, priority scheme, resource monitoring, and resource management. Users submit their serial or parallel jobs to Condor, Condor places them into a queue, chooses when and where to run the jobs based upon a policy, carefully monitors their progress, and ultimately informs the user upon completion.

Pegasus is a tool that can be used in addition to Condor for mapping execution. It is not currently planned for use in the EITS experiment, however it has some capabilities that may be useful in future video experiments so it is listed here for reference.

From the website http://pegasus.isi.edu/

"Pegasus (Planning for Execution in Grids) is a workflow mapping engine developed and used as part of several NSF ITR projects (GriPhyN, NVO, and SCEC-CME).Pegasus bridges the scientific domain and the execution environment by automatically mapping the high-level workflow descriptions onto distributed infrastructures such as the TeraGrid, the Open Science Grid, and others. Pegasus enables scientists to construct workflows in abstract terms without worrying about the details of the underlying CyberInfrastructure. Pegasus provides robustness and reliability through dynamic workflow remapping. Pegasus automatically manages data generated during workflow execution and capturing their provenance information. Pegasus is used in a variety of scientific applications ranging from astronomy, biology, earthquake science, gravitational-wave physics and others."